{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Copyright(c) 2025 NVIDIA Corporation. All rights reserved.\n",
    "\n",
    "NVIDIA Corporation and its licensors retain all intellectual property\n",
    "and proprietary rights in and to this software, related documentation\n",
    "and any modifications thereto.Any use, reproduction, disclosure or\n",
    "distribution of this software and related documentation without an express\n",
    "license agreement from NVIDIA Corporation is strictly prohibited."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# View Grafana dashboard \n",
    "\n",
    "This notebook provides step-by-step instructions for viewing the logs and traces for the Digital human interactions. The objective of the assignment is to gain familiarity with the Grafana dashboard and to correlate logs and traces. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ensure that the Tokkio application is running. \n",
    "Use the following command to check if all pods are in a running state. \n",
    "\n",
    "Note: You will need to select the python environment to run the notebook for the first time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get po -n app"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch the Grafana dashboard\n",
    "\n",
    "Run the script below to obtain the URL needed for the Grafana dashboard. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grafana dashboard URL: http://10.176.223.30:32300\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Extract the server IP address from the Kubernetes config\n",
    "server_ip=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' | sed 's|http[s]*://||' | cut -d':' -f1)\n",
    "\n",
    "# Ensure a valid server IP was found\n",
    "if [[ -z \"$server_ip\" ]]; then\n",
    "  echo \"Error: Unable to extract server IP from Kubernetes config.\"\n",
    "  exit 1\n",
    "fi\n",
    "\n",
    "# Append port 30180 to the extracted IP\n",
    "grafana_url=\"${server_ip}:32300\"\n",
    "\n",
    "# Output the result\n",
    "echo \"Grafana dashboard URL: http://$grafana_url\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "* The dashboard can be launched from the browser within the desktop. \n",
    "* Use the IP address obtained from the previous step and in a browser launched on the desktop.\n",
    "* The default username/password is admin/admin.\n",
    "* You can skip re-setting the username/password for the purpose of this assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View logs\n",
    "\n",
    "* Navigate to HOME > Explore\n",
    "* Select Loki as the datasource\n",
    "* Select a valid time range (e.g., last 6 hours)\n",
    "* Select label filters (e.g., namespace = app)\n",
    "* Narrow the search down by adding the criteria of interest in line contains section (optional). In the screenshot below, the filter for a particular stream id of interest if used as an example. \n",
    "* Run the query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Loki View](pictures/loki_view.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View traces\n",
    "\n",
    "A trace represents the entire journey of a request as it moves across all the microservices in the application. Traces improve the visibility of the overall workflow, help expedite triaging and troubleshooting effort and help track the end-to-end latency. \n",
    "\n",
    "A span is a unit of work within a trace, and has a start time, duration and operation name. In this section, we will view the tokkio root span (i.e. how long a transaction lasts) using the dashboard. Note that you need to close the UI window for interacting with Tokkio application if you want to see the root span for the current/latest transaction/stream ID.  \n",
    "\n",
    "* Navigate to HOME > Explore\n",
    "* Select Tempo as the datasource\n",
    "* Select a valid time range (e.g., last 6 hours)\n",
    "* Select tag filter (e.g., span.stream_id = 'xxxxxxxxxxx' ).\n",
    "* Run the query. \n",
    "* Click on the trace of interest to open the trace view on the right, as shown in the snippet below. \n",
    "\n",
    "![Trace View](pictures/trace1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check the latency\n",
    "\n",
    "In the snippet shown above, we have highlighted the RAG transaction latency of the pipeline for various queries that were made while interacting with the bot. Note that in this particular case, the bot pipeline was configured to use the NvidiaRAGService. \n",
    "\n",
    "One can get similar traces for NvidiaLLMService as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "You have successfully familiarized yourself with the Grafana dashboard for the Digital Human application. Feel free to view various dashboards and spend some more time here!  🎉"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
